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through the experiences of an external evaluation team. Some have called 
evaluating evaluators and their work evaluation auditing, but it could also 
be viewed as a form of meta-evaluation. At the request of the Director of 
Research and Evaluation for the n ESU 18" (named for a county administrative 
unit) evaluation team in Lincoln (Nebraska) , three directors of research and 
evaluation in other school districts (DREs) formed an external visitation 
team (EVT) to conduct a meta- evaluation of the Evaluation Team of the 
initiative (ET) . The ET has a unique relationship to the Lincoln Public 
Schools because it is housed in the same building but is administratively 
separate, reporting to a county administrative unit. This gives the ET the 
advantage of being independent, and less subject to pressure to conduct 
evaluations that create positive public relations for the school district. By 
the same token, their independence lends credibility to their studies and 
reports. Challenges exist, at least theoretically, in that the program staff 
members who receive the evaluations are under little obligation to pay 
attention to them. This results in a need for the ET to work closely with 
school district staff to demonstrate the value of their contribution to 
instruction program quality. This they have accomplished admirably. The 
Lincoln Public School staff viewed the ESU18 evaluators as intelligent, 
thoughtful, creative problem solvers and facilitators. These evaluators were 
seen as adding value to the educational system. Results also show that using 
the EVT to conduct an evaluation of the evaluators is both viable and useful. 
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Evaluating the Evaluators: the external evaluator’s perspective 



Joe B. Hansen 

Colorado Springs School District Eleven 



This symposium poses a significant question for all evaluation practitioners - Who evaluates the 
evaluators? This venerable question has been the subject of discussion and AERA presentations 
for nearly three decades. It is rife with implications for maintaining a measure of quality control 
over the evaluation enterprise, just as that enterprise purports to do with the educational 
programs and services it evaluates. Some have dubbed evaluating evaluators and their work 
evaluation auditing but it could also be viewed as a form of meta-evaluation. 

The earliest use of the term “meta-evaluation” may have been by Scriven (1972) in An 
Introduction to Meta-Evaluation, where he refers to meta-evaluation as a “second order 
evaluation,” with both theoretical and practical dimensions. Stufflebeam (1978) defined “meta- 
evaluation” as the evaluation of evaluations. He distinguished between “primary evaluations,” 
those that are the subject of meta-evaluations and the meta-evaluations themselves. His seminal 
work in this area proposed standards and guidelines for judging the quality of evaluations that 
foreshadowed the eventual development of The Standards for the Evaluation of Educational 
Programs, Projects and Materials (Joint Committee for Educational Evaluation, 1981.) 

Wentling and Klit (1973) described the meta-evaluation of a statewide system for evaluating the 
occupational education programs of local education agencies in Illinois. Their work extended the 
concept of meta-evaluation to that of evaluating not only the evaluations, but the system 
responsible for those evaluations, including the personnel. I discussed a process for quality 
control in Title-I evaluation in 1977, based on the role of an evaluation auditor, overseeing the 
evaluations of Title-I programs (Hansen, 1977.) In that article, I laid out a series of issues to be 
addressed by anyone attempting to evaluate the work of other evaluators. More recently, meta- 
evaluation has been incorporated into the Program Evaluation Standards(Joint Committee on 
Standards for Educational Evaluation, 1994) as a standard (A12.) Briefly, the standard on meta- 
evaluation states: 

“The evaluation itself should be formalively and summatively evaluated against these 
and other pertinent standards, so that its conduct is appropriately guided anti on 
completion, stakeholders can closely examine its strengths and weaknesses. " (P. 259) 

The process described in the Novak paper and here, applies the concept of meta-evaluation to an 
evaluation unit and the services it provides to a public school system. Whereas meta-evaluation 
usually focuses on an individual evaluation study or process and asks questions about its adequacy 
as to its methods and procedures, data interpretation, conclusions, recommendations and so on; 
the process engaged in by the external visitation team (E.T.) for ESU 18, examined the broad 
range of services provided by the ESU evaluation unit staff and sought to provide formative 
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Hansen, External Perspective 

evaluation recommendations that the ESU 18 Evaluation Team (ET) could use to refine and 
improve its evaluation services to Lincoln Public Schools (LPS). 

As one of the external evaluators in this process, I will address the following questions: 

• What was the role of the external visitation team and what did the team do during the 
visitation? 

• Were there any restrictions placed on the visitation team? If so, did the restrictions bias 
the evaluation results/findings? 

• What did the external team learn during the evaluation and how were those findings 
communicated? 

• What issues and questions should have been addressed in the evaluation, but were not? 

• What did the external team learn about this process that would be of value to others 
considering this type of evaluation? 

I will address each of these questions sequentially. 

What was the role of the external visitation team and what did the team do during the 
visitation? 

The EVT had the purpose of validating or verifying the self study, previously conducted by the 
ESU 18 Evaluation Team (ET) and providing additional observations and recommendations to 
help the ET meet its goals. In other words this was a formative meta-evaluation of the ET. The 
approach taken by the EVT was to meet with LPS administrative staff, who had received 
evaluation or evaluation support services from the ET and elicit their observations of the ET’s 
performance. Three people, each of whom was a director of research and evaluation (DRE) in a 
school district, comprised the EVT. 1 Each of the three members of the EVT was asked to focus 
on a particular aspect of the ET or its work. One focused on the district assessment program and 
the quality of work done on it. Another focused on the quality of work done on projects. The 
third focused primarily on internal operations and how projects were assigned to the team. 

Methodology 

The methodology used by the EVT consisted of document review, interviews with individual LPS 
administrators, interviews or discussions with groups of administrators and a discussion with the 
ET professional staff. Documents reviewed included: 



\ The three team members were Michael Herrick, Eau Claire, Public Schools; Stephan 
Henry, Topeka Public Schools; and Joe Hansen, Colorado Springs Public Schools. The work 
reported here was performed by the entire team, not just the author. This paper is presented with 
the approval of the other members. 
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Hansen, External Perspective 

• ESU 1 8 Evaluation Team Accreditation Studies Part 1 : Background 

• ESU 18 Evaluation Team Accreditation Studies Part 2: Results and Recommendations 

• A list of Evaluation Team Programs and Projects included in the Self Study 

• 1996-97 ESU Evaluation Team Project Time Log Report 

• 1996-97 Time use Reports 

• 1 995-96 ESU Evaluation Team Project Time Log Report 

• 1995-96 Time Use Reports 

• Improving Assessment Practices in the Lincoln Public Schools: The Final Report of the 
Assessment Study Committee. 

After reviewing these documents the EVT members elected to adopt the five key questions used 
in the Self Study as a framework for our review. Those questions are as follows: 

1. Do current Evaluation Team assignments and past assignments, reflect district priorities? 

2. Does the work done by the Evaluation Team on those assignments conform to the 
generally accepted standards for educational evaluation and does that work meet the needs 
of their clients? 

3. How can the team best support the district over the next two to five years? 

4. Are the principles described in Part 1 of the Self Study still relevant and are the procedures 
followed by the Evaluation Team consistent with these principles? 

5. Does the Evaluation Team have the support staff and resources necessary to be successful 
and are the support staff and resources used efficiently? 

Our EVT conducted interviews and group meetings with the following individuals and groups. 

The Associate Superintendent for Instruction, Instructional Division Directors for Staff 
Development, Special Education, Student Services, Library/media services; the Administrative 
Assistant for Curriculum and Instruction; Consultants for Math, Business Education, Early 
Childhood, Elementary Reading and Language Arts; the Curriculum Director, Externally Funded 
Grants staff and Principals of four Schools that had received grants for assessment and 
instructional improvement. They included a high school, a middle school and two elementary 
schools. 

The three EVT reviewers also met with the ET professional staff. At the end of the visit our EVT 
conducted an exit interview and provided our findings, commendations and suggestions for 
improvement. 

Were there any restrictions placed on the visitation team? If so, did the restrictions bias 
the evaluation results/findings? 

The only restrictions we experienced were those that derived from the schedule and logistics of 
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the study. The EVT had an evening meeting with the Leadership Team of the ESU on the first 
night, to receive an orientation to our task. We then had one day to collect data and prepare our 
recommendations. With more time, we might have had an opportunity to pursue in greater depth, 
some of the issues and questions that arose naturally out of our interviews and discussions. We 
could have also sought out input from other staff who were not preselected for us. Although it is 
doubtful that such input would have varied significantly from that which we received, it is possible 
that we might have heard some constructive criticism of, or possibly even dissatisfaction with the 
ET. I don’t believe however, that the lack of opportunity for such follow up resulted in a bias. 
More time would have enabled us to develop more detailed and tightly focused recommendations 
and to validate the findings. 

What did the external team learn during the evaluation and how were those findings 
communicated? 

Following the five study questions they gave us, we learned much about how ET priorities were 
set, the quality of the ET staff, applicability of the Evaluation Standards in the ET’s work, the 
level of satisfaction clients felt with the ET’s work, the advantages of the unique organizational 
relationship of the ET to LPS, the role of ET staff, and the need for assessment planning. 

Priorities. The ET Director (Dr. Novak) had established a clear process for arriving at priorities 
and setting goals. He had also established a procedure for tracking and monitoring staff time by 
project. Key administrators were familiar with and supportive of the ET’s priorities. These same 
key administrators felt involved in the ET’s priority setting process and were sympathetic to the 
ET for not having more time to devote to providing consulting assistance, which was viewed by 
the curriculum and instruction staff as highly beneficial. Some LPS directors felt that the ET 
doesn’t really get to set their own priorities; that instead they are set for them. State and federal 
requirements come first, followed by whatever the superintendent and Board want, then they 
allocate the rest of their time to projects. The ET staff felt this was an accurate depiction. 

It was obvious to the EVT that the ET and LPS engage in systematic strategic planning in which 
assessment projects are designed to contribute to the district mission and strategic plan for 
instruction. Although the ET reported struggling with limited resources and that district priorities 
were not always clear, the district consumers of their services were very pleased with their work. 
These customers reported to the EVT that they understood that the ET had multiple audiences 
and had to allocate their limited resources among many potential project areas. We heard no 
criticism regarding the manner in which the ET allocated staff time to projects. Despite the 
commitment “up-front” to a heavy project load, the visiting team heard repeatedly that the 
Evaluation Team routinely responds to on-the-fly requests for custom data analysis and does so in 
a very timely manner. The recipients of the ET’s services were pleased with the service they had 
received and praised the ET for the quality of its work. 
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Quality of staff. The ET consists of a highly trained and highly experienced staff who have 
earned the respect of district colleagues. They are called upon regularly to support program 
planning and development efforts for the district and have had a positive impact on the district’s 
effectiveness by promoting a research-based scholarly approach to district planning and program 
development. Comments heard about the ET in interviews included “They really help us improve 
instruction,” “They are good at making suggestions,” and “They are there when I need them.” 

LPS administrators recognized the evaluators as seeing the “big picture.” That is, they see the 
need to visualize ultimate outcomes of programs, and see connections to other programs within 
the district. In addition, they viewed them as objective, unbiased, and expert data collectors. 

They were considered able to analyze a problem and communicate results of an analysis clearly to 
staff and to parents. They were also recognized for being honest in their interpretation of data. 

ET members are not compromised into presenting findings that are “acceptable” as opposed to 
accurate. In short, evaluators lend credibility and accountability to program development. 

Evaluation Standards. LPS administrators were not explicitly aware of the Evaluation 
Standards, but we listened for evidence of their application in the work of the ET. We heard 
respondents mention two concepts repeatedly that implied the evaluation standards of utility and 
accuracy were highly applicable to the work of the ET. The greatest utility by far, offered to the 
LPS by the ET is that of helping program staff think more clearly about their programs and plan 
more effectively. LPS staff attributed this, in part, to the ET’s focus on outcomes and effects and 
their search for evidence of the outcomes. Because of the involvement of an evaluator, they 
asked and attended to research and evaluation questions during the conduct of program 
evaluations and program development activities. Evaluators are valued because they “ask the 
questions that need to be asked” and they question how data proposed for an evaluation can best 
be used. One curriculum consultant commented that evaluators “define purpose and direction” 
for projects and programs, and “help us be more thoughtful and purposeful.” Evaluators are 
considered key players that provide significant leadership because of the way they think. 
Evaluators think in terms of outcomes and effects, and evidence needed to determine outcomes. 
The evidence from interviews supported the belief that the LPS was a research-based and data- 
driven district, largely because of the work of the ET. 

Further evidence of the utility of evaluation studies conducted by the ET, expressed by staff, is 
that the culture in the Lincoln Public Schools is that of a “research-based district.” There appears 
to be a culture in the district that appreciates research and evaluation for providing programs with 
a focus and that staff supports by asking, “what does the research tell us.” The ET director meets 
with curriculum directors monthly to review research and discuss upcoming projects. 

Respondents felt that this kind of regular contact helps the district to keep evaluation central to 
program planning and implementation. 
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Organizational relationship of the Evaluation Team to Lincoln Public Schools. All 

respondents acknowledged that the ET has a unique organizational relationship with the LPS. 
Because ESU 18 is technically an external organization, evaluations conducted by the ET are 
external evaluations. However, because of the way evaluations are conducted, with evaluators 
functioning as valued consultants and professional colleagues, the evaluations are considered as 
internal. Everyone we talked to indicated that the relationship of the ESU and the district is ideal 
for evaluation. When perceptions by the public or external funding agencies suspect bias, 
program leaders legitimately claim the evaluations are independent and external. However, all 
evaluations are conducted with a high degree of collegiality and teaming. This collegial and 
participatory approach to evaluation is advantageous in that the evaluators understand district 
programs and have a working knowledge of the district as a whole. Also, their continuity in 
conducting evaluations in the district gives them a “big picture” perspective that even Directors 
and Curriculum Consultants may not have. This closeness of the evaluation function and program 
operations does not seem to jeopardize the accuracy and validity of data or the honesty with 
which those data are reported. It also aids in the implementation of study findings and 
recommendations. 

Members of the ET indicated that they feel even more independent than a private consultant might 
be because their jobs are secure regardless of their evaluation findings. They feel no pressure to 
produce positive results for the purpose of gaining repeat business. 

Role of the evaluator. The ET members and most users of evaluation studies see the role of 
evaluator as providing direction for the evaluation in collaboration with the rest of the team, and 
being exclusively responsible for conducting the actual study. For the most part, their 
responsibility ends with the delivery of an evaluation report. The program staff is responsible for 
sharing the results and recommendations of the study. However, the division of those roles and 
responsibilities is not clear cut. ET staff considers this a strength. Program staff frequently call 
on evaluators to assist in implementation. Evaluators do this by explaining and interpreting 
findings to various groups and stakeholders, providing staff development to relevant groups and 
by consulting with directors and consultants on proper implementation. Ultimately, however, 
program staff is accountable for insuring proper implementation. Directors and Curriculum 
Consultants were quick to indicate that the evaluation role makes program success more likely. 



The primary role that evaluators play was described as that of a consultant. This works very well 
because it gives the evaluator an equal position on a curriculum or project team. The project 
team is lead by the person responsible for the program, curriculum area, or the project. Although 
this may appear as an unconventional role for an evaluator, it seems to work well for the ET and 
LPS. 
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Need for an Assessment Plan. Increased demands for assessment driven reform have, in recent 
years, resulted in a shift in focus of staff time and energies from traditional program evaluation 
studies to a far greater emphasis on assessment related consultation. It appeared to us that the 
time had come for the ET to help the LPS to develop a comprehensive assessment plan to address 
current and future assessment needs. The ET staff supported this view. 

Communication of Results 

We communicated our EVT’s findings in two ways. First, through an exit interview meeting with 
ESU leadership and key LPS administrators. In that meeting we shared our observations and 
preliminary recommendations. Second, each EVT member had the responsibility for writing a 
section of a report we would remit to the ESU. In addition to writing my section, I acted as the 
editor, collecting the separate pieces via e-mail, editing them and developing an integrated 
document from them. Each EVT member was then e-mailed a draft to review and edit further. I 
incorporated this input into the final report document and submitted it to the ESU. 

What issues and questions should have been addressed in the evaluation, but were not? 

Given the constraints of time imposed on this review or meta-evaluation, the EVT had all it could 
handle in addressing the questions that were provided. An issue that emerged naturally from the 
process was that of how the ET might leverage or extend its limited resources to increase its 
effectiveness. Our recommendations dealt with this only tangentially, by suggesting more ET 
involvement in staff development, especially for principals in the interpretation and use of data for 
instructional improvement, and providing some additional funding to the ET for outsourcing some 
projects. 

What did the external team learn about this process that would be of value to others 
considering this type of evaluation? 

The evaluation process was loosely structured around the Self Study previously conducted by the 
ET. Time constraints precluded an adequate amount of planning time for the EVT members to 
develop and field test data collection protocols. Having such protocols would have made the data 
collection effort more systematic and better focused. 

The EVT could also have benefited from a better a-priori understanding of the relationship of the 
ET to each of the administrative units and groups we met with and individuals we interviewed. 
Again, this is a function of time. With the availability of one more day, we could have spent more 
time planning together and discussing our approach with the ET director, prior to beginning data 
collection. This would have helped us to develop a standard set of core questions to guide our 
investigations. Additional time beyond that would have been helpful in enabling us to conduct 
follow-up interviews where we felt a need for clarification, or to pursue a particular issue in 



7 



O 

ERIC 



9 



Hansen, External Perspective 

greater detail. For those considering an evaluation of the evaluators I would suggest the 
following guidelines. 

1. Select the team members to represent areas of expertise relevant to the issues and problems 
that are most important to the study. Also, make the team large enough to enable sufficient 
access time to key respondents. 

Our team consisted of three relatively senior DREs. This was a marginal number for the large 
number of LPS staff we needed to interact with. Fortunately, our backgrounds afforded us some 
diversity of perspective roughly consistent with the roles we were asked to assume. It would 
have been helpful to our team to have a person whose professional background and interests were 
more instructionally oriented. A curriculum director or assistant superintendent for instruction 
would have added a valuable perspective to our work. 

2. Provide the external team with enough time together prior to beginning data collection so 
that they can work together to develop a set of core questions they will ask of all groups and 
individual interviewees, and some more specific questions for specific groups. 

A reasonable procedure for getting ready to conduct the data analysis might be as follows. 

A. The local on-site coordinator of the study should conduct an orientation meeting 
with the external team. This meeting should last a minimum of four hours and 
should have as its objectives: 

1 . developing a common understanding of the key issues to be addressed, the 
audiences involved and the relationships of those audiences and individuals 
to the evaluation unit being evaluated. 

2. cooperatively developing a set of core questions to be asked of most, if not 
all respondents. 

3. developing sub-sets of questions for groups and individuals based on their 
professional roles and their specific relationships to the evaluation unit. 

4. reviewing, critiquing and refining the questions together. 

5. arriving at an agreement on how interviews would be conducted for 
individuals and groups so that a well understood process is consistently 
used. 

B. The visiting team should become familiar with the Evaluation Standards and use 
the Standards in formulating the core questions. 

C. The team should identify a team leader who will coordinate the work and serve as 
a process facilitator for team discussions. 
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D. The on-site coordinator should conduct a general orientation meeting with the staff 
who will be involved in the study and the visiting team. In this meeting, the 
coordinator should explain the purpose of the study and introduce each of the 
external team members, describing their qualifications and roles in the study and 
the staff members. This will help the visiting team to establish their credibility and 
will make it easier for them to establish rapport with respondents. 

E. The schedule must include enough time so that the external team can meet for 20 
to 30 minutes twice each day, to check impressions with one another, ask 
questions of the coordinator, reflect on the data and decide on follow up needs. 

F. An adequate amount of time must be allowed for data collection, review of notes 
and group time for the external team to develop their recommendations and review 
them together prior to the exit interview. The amount of time necessary for this is 
obviously a subjective judgement. However, it is likely that with the initial 
orientation meeting, the orientation with on-site staff who will be interviewed and 
the follow-up time needed for clarification and recommendation development, at 
least two full days on-site will be needed. 

Summary and Conclusions 

At the request of the Director of Research and Evaluation for the ESU 18 Evaluation Team, in 
Lincoln, Nebraska, three DREs formed an external visitation team or EVT to conduct a meta- 
evaluation of the Evaluation Team of the ESU. The ET has a unique relationship to the LPS, 
since it is housed in the same building, but is administratively separate, reporting to the ESU, a 
county administrative unit. This creates both challenges and opportunities for the ET staff in their 
relationship with the LPS administrators they serve. The ET has the advantage of being 
independent, therefore they are less subject to pressure to conduct evaluations that create positive 
public relations for the LPS but fail to provide the constructive criticism needed for improvement. 
By the same token, their independence lends credibility to their evaluation studies and reports. 
Challenges exist, at least theoretically, in that the LPS program staff members who are recipients 
of the evaluation services are under little obligation to pay any attention to them, except in the 
case of the federal and state mandated program evaluations. This results in a need for ET to work 
closely with the LPS staff to demonstrate the value of their contribution to instruction program 
quality. This they have accomplished admirably, resulting in a harmonious and symbiotic 
relationship and an increasing demand for their services. 

LPS staff viewed ESU18 evaluators as intelligent, thoughtful, creative, problem solvers and 
facilitators, whose contributions improved the quality of district programs. Evaluators’ 
participation was sought in planning, implementation and evaluation of district programs. They 
were uniformly viewed as adding value to the LPS educational system. 
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The process of conducting the evaluation of the evaluators was highly instructive to those of us 
who participated in it. We learned that a highly effective model for delivering evaluation services 
has been established in an out of the way Midwestern community, and that it has heretofore been 
a well-kept secret. I recommend that every public school evaluator who wants to see how 
evaluation can have a positive impact on public education, in a collaborative, non-traditional way, 
should schedule a visit with the Evaluation Team at ESU 1 8. 

We also learned that the process of using an external visitation team or EVT to conduct an 
evaluation of the evaluators is both viable and useful. In this paper, I have outlined suggestions 
for structuring a process to enhance the effectiveness of the EVT. I also recommend to anyone 
considering this type of work, that they follow the Evaluation Standards. I encourage evaluators 
to form teams and visit each other’s shops, conducting this form of collegial review. I would 
welcome feedback on the process I have outlined, in the hope that its refinement will lead to the 
development of a model for the evaluation of evaluation units and lead evaluators to the 
continuous improvement of their work. 
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University of Maryland 
1129 Shriver Laboratory 
College Park, MD 20742-5701 

Tel: (800) 464-3742 
(301) 405-7449 
FAX: (301) 405-8134 

March 20, 1998 encae@ericae.net 

http://ericae.net 

Dear AERA Presenter, 

Congratulations on being a presenter at AERA 1 . The ERIC Clearinghouse on Assessment and Evaluation 
invites you to contribute to the ERIC database by providing us with a printed copy of your presentation. 

Abstracts of papers accepted by ERIC appear in Resources in Education (R1E) and are announced to over 
5,000 organizations. The inclusion of your work makes it readily available to other researchers, provides a 
permanent archive, and enhances the quality of RIE. Abstracts of your contribution will be accessible 
through the printed and electronic versions of RIE. The paper will be available through the microfiche 
collections that are housed at libraries around the world and through the ERIC Document Reproduction 
Service. 




We are gathering all the papers from the AERA Conference. We will route your paper to the appropriate 
clearinghouse. You will be notified if your paper meets ERIC's criteria for inclusion in RIE : contribution 
to education, timeliness, relevance, methodology, effectiveness of presentation, and reproduction quality. 
You can track our processing of your paper at http://ericae.net. 

Please sign the Reproduction Release Form on the back of this letter and include it with two copies of your 
paper. The Release Form gives ERIC permission to make and distribute copies of your paper. It does not 
preclude you from publishing your work. You can drop off the copies of your paper and Reproduction 
Release Form at the ERIC booth (424) or mail to our attention at the address below. Please feel free to 
copy the form for future or additional submissions. 



Mail to: AERA 1998/ERIC Acquisitions 

University of Maryland 
1129 Shriver Laboratory 
College Park, MD 20742 



This year ERIC/AE is making a Searchable Conference Program available on the AERA web page 
(http://aera.net). Check it out! 



Sincerely, 




Lawrence M. Rudner, Ph.D. 
Director, ERIC/AE 




l If you are an AERA chair or discussant, please save this form for future use. 
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The Catholic University of America 




